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(54) METHOD AND DEVICE FOR DISPLAYING AND EDITING 

(57)Abstract: 

PURPOSE: To provide a system which edits a multimedia 
project on a display picture. 

CONSTITUTION: The computer display picture presented to 
a user provides three different views of multimedia 
information. The first view is a player 1 1 similar to a virtual 
video recorder accompanied with a monitor. The second 
view is a story board 13 which resembling a conventional 
story board very much. The third view is a strip viewer 15 
which presents a view based on the time base of multimedia 
information to the user. There is an epoch-making screen 
tool called a 'splinter' on the strip viewer display part, and it 
is used to edit the start point and the end point of a 
displayed multimedia track. 
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* NOTICES * 



JPO and INPIT are not responsible for any 
damages caused by the use of this translation. 

1. This document has been translated by computer. So the translation may not reflect the original 
precisely. 

2. **** shows the word which can not be translated. 
3. In the drawings, any words are not translated. 



DETAILED DESCRIPTION 



[Detailed Description of the Invention] 
[0001] 

[Industrial Application]Especially this invention relates to the method and device which provide 
a user with the intuitive method for editing the multimedia information which displays 
multimedia information on a display screen, and a display screen is shown about the field of a 
multimedia display and an editing device. 
[0002] 

[Description of the Prior Art]Multimedia combines with a single work the media from which two 
or more molds differ. For example, the multimedia work could consist of video information, 
audio information, figure overlay, and combination of animation. The opportunity to develop a 
multimedia work, and for a computer to be used in order to display has increased all the more 
because the computer is suitable for combining such various media types. Since creation of a 
multimedia production is relatively [ to manufacture of video ] similar, on the occasion of 
creation of a multimedia production, the technique of video is used in many cases. For example, 
standard video editing tools, such as a storyboard and a videocassette recorder for edit, are used 
in many cases as a multimedia development tool for assisting with creation of a multimedia 
production. However, if a computer is used, the new interactive mode tool with which creation 
and edit of a multimedia production are assisted can be created. Probably, as for those new 
interactive mode tools, it will be ideal that both learning and using it constitute an easy intuitive 
user interface. 
[0003] 

[Problem(s) to be Solved by the Invention]Therefore, the purpose of this invention is to provide 
the method and device which display multimedia information on a computer display screen. 
Another purpose of this invention is to provide a user with the method and device into which it 
enables it to edit the multimedia information displayed on the display screen by an intuitive 
method. Another purpose of this invention is to show a user the intuitive method which extends 
and carries out trimming of the track of multimedia information by displaying multimedia 
information as a strip with the tool for edit used in a strip extension or in order to carry out 
trimming. 
[0004] 

[Means for Solving the Problem]The above-mentioned purpose and the other purposes are 
attained by an original method and a device of this invention. A method and a device of this 
invention comprise a system into which a multimedia project is edited in a display screen. A 
computer display screen which a user is shown forms three different views of multimedia 



information. The 1st view is a player similar to a virtual VCR accompanied by a monitor. The 
2nd view is a storyboard which is very well alike in the conventional storyboard. The 3rd view is 
a strip viewer which shows a user a view based on time line of multimedia information. A strip 
viewer display is a screen tool called "SUPURTNDA (splinder)." SUPURENDA is used in order 
to edit the starting point and an end point of a multimedia track. The purpose, the feature, and an 
advantage of this invention will become clear to a person skilled in the art from the following 
detailed explanation. 
[0005] 

[ExampleJThe method and device which display multimedia information and are edited are 
indicated. During the following explanation, in order to make this invention understand 
thoroughly, the term of explanation specific for convenience is mentioned, but probably, it will 
be clear to a person skilled in the art that such a specific detailed matter is not required, in order 
to carry out this invention. In order not to make this invention unclear unnecessarily depending 
on the case, the gestalt of a block diagram may show a well-known circuit and device. 
[0006] Reference of drawing 1 shows the multimedia editor display screen using instruction of 
this invention. A multimedia editor display screen shows a user three the "views" of multimedia 
information. The player 1 1 provides the 1st view of multimedia information. The player 1 1 is 
designed simulate operation of the videotape recorder for edit, therefore it is used in order to 
observe and record a live video. A user "records" video by using the player 1 1 and identifying, a 
start video frame (inch-point), i.e., a "yne point", and an end video frame (out-point), i.e., an "out , 
point." The segment which comprises the continuous video frame which was made such and 
specified is called the "shot" of video. While the video player is not operating, the frame of the 
beginning of the present video selection is displayed on an observation area. 
[0007]The storyboard 13 of drawing 1 provides the 2nd view of multimedia information. The 
storyboard 13 gives a user the view of the multimedia information of "a higher level" by 
displaying all the shots of video as a single icon so that the shot 5 may show from the shot 1 . The 
user can record, delete and copy all the shots of video by using the storyboard 13, and selecting 
the icon which expresses a shot with cursor, next choosing desired editing work from a menu. 
[0008]Or the storyboard 13 can be used in order to edit high-level video structures, such as a 
"scene" or an "act." These terms are borrowed from the theatrical world, and they are used in 
order to define the video of a higher level, or a group division of a film. For example, the 
producer/artist who creates a multimedia production may desire to collect a series of video shots 
into one a "scene." If one example is given, a series of shots which photoed the man and woman 
who are talking in the French cafe with variously different camera angle will be able to be 
summarized as a scene of a "French cafe." Then, the aggregate of a scene can be packed into the 
"act" of a higher level. If it continues about a previous example, a series of scenes including the 
scene where man and woman meet first several times will be able to be summarized as an act of 
"encounter of man and woman." One example of the layered structure of a shot is indicated to be 
an act considered that it can use drawing 2 in order that a producer may compose a multimedia 
production, and a scene. 

[0009]However, the storyboard 13 of drawing 1 does not impose the strict layered structure 
which a producer has to use. Therefore, this storyboard 13 can be used for editing with the 
arbitrary layer levels which a user desires. For example, in order for a producer to complete the 
mechanics of one specific scene, the storyboard 13 may be used on a "shot" level. It can raise to 
a "scene" level and the overall feel of the mood of the multimedia production can be held. 
[0010]If two views of these starts, i.e., the player 1 1 and two views of the storyboard 13, are 



combined, the functionality of the videotape recorder for edit and an ordinary storyboard will be 
obtained. Thus, the tool of the conventional video edit studio is given to a user. Therefore, if it is 
a user expert in the conventional video editing tool, it is possible to learn the directions of the 
multimedia editor explained above to the inside of a short time, and to use it. 
[001 l]The 3rd view of the multimedia information which the multimedia editor display screen of 
drawing 1 presents is the strip viewer 15. Details look at the contents of the media track, so that 
this view was impossible at in advance. If the strip viewer 15 is used, a user is a time-line 
method and can observe multimedia information with the level of resolution which can moreover 
be adjusted. The strip viewer 15 expresses multimedia information as the most detailed level 
(namely, the highest resolution) as "atomic" piece which cannot be decomposed. For example, 
when a user observes video information on the most detailed level (namely, the highest 
resolution) with the strip viewer 15, the frame of all individual videos is displayed. 
[0012]If an example is given, the case where the strip viewer 15 of drawing 1 is set as the highest 
resolution so that each video frame may be displayed, respectively will be considered. The 
display is similar to the strip of the motion-picture film in which each frame has an original 
frame number. As for a time scale, i.e., a "resolution" scale, when displaying multimedia 
information, multimedia information is shown caudad. In drawing 1 , since it is the highest 
resolution, the resolution scale is enumerating continuous frame numbers. Resolution can be 
adjusted so that the frame in the every every 5th 30th 100th [ every ] or the frame of every 
multiple frames of other may be displayed. Therefore, the user can scroll one by one by "low 
resolution" using the well-known scroll bar 17, and can find out a rough area of interest, next can 
zoom in the highest resolution, and can inspect the field in detail. 

[0013]Three views of the multimedia information shown in drawing 1 have a dialog mutually, 
and can influence each other. If "caret" 14 used in order to direct the inserting point in the 
storyboard 13 are moved, since the position of this caret is reflected, a strip viewer will update 
oneself. This is shown in drawing 1 and the caret 14 of the storyboard 13 is set up between the 
1st shot and the 2nd shot. This position of the caret 14 is reflected into the strip viewer 15, and 
the strip viewer shows the boundary of the frame of the 1st shot and the 2nd shot. The innovative 
tool called "SUPURTNDA (splinder)" 1 is between the frame of the 1st shot, and the frame of the 
2nd shot. It can be used in order to adjust a boundary, as SUPURINDA 1 is used in order to mark 
the boundary of a shot, and it is explained below. Similarly, the adjustment performed in the 
field of the strip viewer 15 also affects the field of the storyboard 13. For example, if the caret 19 
of the strip viewer 15 is moved, since a motion of the caret 19 in a strip viewer is reflected, the 
caret 14 of the storyboard 13 will be updated. The player 1 1 can be used and the video data 
chosen with either the storyboard 13 or the strip viewer 15 can be reproduced. In order to specify 
the segment which should be edited by setting up the starting point and an end point while the 
player 1 1 shows video, it is also possible to use the player 1 1 . 

[0014]The strip viewer 15 is not limited to a display and edit of video information. The interface 
of the strip viewer 15 is common enough so that any numbers of media tracks with which molds 
differ may be shown. Another example of the strip viewer which displays the three different 
tracks 21 of media information, i.e., video, the audio 23, and three tracks of the figure overlay 25 
is shown in drawing 3 . It is also possible to be also able to edit individually, respectively, and to 
edit three media tracks collectively. As a media track otherwise considered, there are a notes 
track, a video special-effects track, a 35-mm slide track, and (in order to emulate a motion) a 
hydraulic actuator track (however, not limited to those tracks). The media information which 
carries out a visual indication to a screen is actually memorized by the related database. The 



database used in order to memorize information is explained still in detail below. 
[0015]Down the three tracks of the multimedia information of the strip viewer of drawing 3 , the 
scale 27 which displays a frame number or a time basis is located. Between each frame 542 and 
frame 543 of three media tracks (21, 23, and 25), the icon called "SUPURINDA" (la, lb, and lc) 
is arranged. This position expresses the boundary of two different shots so that the contents of 
the video frame of the media track 21 may show, being used in order that SUPURINDA (la, lb, 
and lc) may divide multimedia information into two different shots — the media information — 
on the other hand — or it is extensible to the shot of another side. The term of "SUPURINDA" 
was created by combining two terms of " splitter (SPLItter)" and " extender (ExtenDER)." 
[0016]The detailed figure of SUPURINDA 601 is shown in drawing 4 . SUPURINDA 601 is 
displayed between the video frames belonging to two different shots. The terminal frame of the 
1st shot in the left-hand side of SUPURINDA 601 is expressed by the frame 633, and this is 
hereafter called "1st shot." The first two frames of the 2nd shot in the right-hand side of 
SUPURINDA 601 are expressed by the frame 635, and this is hereafter called "2nd shot." The 
icon showing SUPURINDA comprises the head 603, the crus-sinistrum-diaphragmatis part 605, 
and the right trail part 607. Each portion of the head 603 of SUPURINDA, the crus-sinistrum- 
diaphragmatis part 605, and the right trail part 607 is the "hot spot" which can be caught and 
pulled apart with cursor controllers, such as a mouse. For example, what is necessary is just to 
push a switch, in order to report the prehension for SUPURINDA to "catching" and the next by 
moving a mouse until cursor comes to the upper part of SUPURINDA. Then, SUPURINDA can 
be pulled apart by moving a mouse, pushing a switch. When a switch is released, since execution 
of edit finishes and the edit is reflected, it turns out that the view was updated. As for a strip 
viewer, it is preferred to carry out focusing of the view again at the place where forefront edit 
was performed. When the head 603 of SUPURINDA is caught and pulled apart, whole 
SUPURINDA 601 moves, and the shot (633 and 635) of the both sides of SUPURINDA is 
edited. However, when either the crus-sinistrum-diaphragmatis part 605 or the right trail part 607 
is caught and pulled apart, only the leg and a corresponding shaft part move and only the shot of 
the side corresponding to it is edited. 

r00171 Drawing 5 (a) - (c) shows edit of the video track by use of SUPURINDA in detail. 
Drawing 5 (a) In - (c), the video frame in the left-hand side of SUPURINDA is called "the 1st 
shot", and the video frame in the right-hand side of cursor is called "the 2nd shot." Reference of 
drawing 5 (a) shows some figures of the strip viewer. In (a), a user places the cursor 590 on the 
head 503 of SUPURINDA 512, and "holds" the head 503 of SUPURINDA 512 Using cursor 
controllers, such as a mouse. Next, whole SUPURINDA is lengthened to the left by the head 503 
of SUPURINDA 512 so that it may direct with the arrow used since a motion of cursor is 
expressed. The beginning of the shot in the right-hand side of SUPURINDA 512 is extended at 
the same time it trims the end of the shot which is in the left-hand side of SUPURINDA 512 by 
lengthening SUPURINDA 512 to the left by the head. In drawing 5 (a), since a user holds the 
head 503 of SUPURINDA with the cursor 590 and it is lengthening to the left, trimming of the 
frame belonging to the 1st shot is carried out, and the frame belonging to the 2nd shot is 
extended. It means that the term of "trimming" deletes the frame which is not desired. It means 
that the term of "extension" adds a frame to the boundary of a shot. If length of SUPURINDA is 
stopped and SUPURINDA is stopped at a suitable place, a strip viewer will memorize the 
performed edit. In this example, after a user completes edit, a frame is added to one shot, and 
since it reflects that trimming of same number of the frames was carried out from the shot of 
another side, a database is updated. 



[0018]Drawing_5_(b) shows how SUPURINDA can be used, in order to trim a frame only from 
the end of the 1st shot. In (b), a user holds the crus-sinistrum-diaphragmatis part 505 of 
SUPURINDA with the cursor 590, and he lengthens the crus-sinistrum-diaphragmatis part 505 
and its shaft part 504 to the left so that it may direct with the arrow used since a motion of cursor 
is expressed. Trimming is carried out so that the end of the 1st shot may see and show a frame by 
the eye, as dragging of the shaft part 504 is carried out to the crus-sinistrum-diaphragmatis part 
505 of SUPURINDA along with the frame of the 1st shot, the crus-sinistrum-diaphragmatis part 
505 of SUPURINDA - the - if it is alike shaft part 504, a motion is stopped and SUPURINDA 
is stopped at a suitable place, a strip viewer will memorize the edit. Then, since it reflects that 
trimming of some frame was carried out from the end of the 1st shot, the multimedia database in 
this example is updated. When edit is completed, the "hole" between two shots is lost and 
SUPURINDA is returned to the original shape. 

F00191 Drawing 5 (c) shows how SUPURINDA can be used, in order to add the frame 
beforehand cut into the 1st shot. In (c), a user holds the crus-sinistrum-diaphragmatis part 505 of 
SUPURINDA with the cursor 590, moves it to the right, and is extending the end of the 1st shot. 
It is a wrap temporarily about the frame which the frame "was cut" beforehand from the end of 
the 1st shot appears, and belongs to the 2nd shot as the crus-sinistrum-diaphragmatis part 505 
and the shaft part 504 of SUPURINDA are moved to the right. If the crus-sinistrum- 
diaphragmatis part 505 of SUPURINDA and a motion of the shaft part 504 are stopped and 
SUPURINDA is stopped at a suitable place, the edit is memorized, and since it reflects that the 
frame was added in the end of the 1st shot, a database will be updated. If the frame of the 1st 
shot is moved to the left when edit is completed, the frame with which the 2nd shot was covered 
temporarily will appear, and SUPURINDA will return to the original shape. 
[0020] Instead of the left leg and shaft part, operation which has influence in the end of the 1st 
shot shown in drawing 5 (b) and (c) also by the 2nd shot beginning can be similarly performed 
by using the right leg and shaft part. For example, when the leg on the right of SUPURINDA is 
caught and SUPURINDA is moved to the right along with the frame of the 2nd shot, trimming 
will be carried out from a video frame from the start of the 2nd shot. When similarly the right 
trail part of SUPURINDA is held and SUPURINDA is moved to the left along with the frame of 
the 1st shot, the frame cut previously will be added at the beginning of the 2nd shot. 
[0021]When a user "edits" multimedia information on a display screen using SUPURINDA of 
this invention, "edit" on a screen is used and the relation database which memorizes multimedia 
information is updated. One example of an usable digital video database is shown in drawing 6 . 
However, the edit-on screen technique explained here may use other database structures rather 
than is limited to this specific real original form voice of a multimedia database. 
[0022]Reference of drawing 6 shows the real original form voice of the usable digital video 
database. There are the three rectangles 600,605 and 610 showing a compression digital video 
data file located in a line in the center of a figure. Each video-data file contains the compression 
digital data which specifies the frame of video. The tree data structure 615 showing the 
multimedia production which edited is above a video-data file. Since a tree structure is very close 
to the hierarchy whom the producer of video or a film is using in order to edit video and a film 
project as shown in drawing 2 , it is preferred. 

[0023]There are the data structures 620, 625, 630, 635, 640, and 645 which specify the segment 
of the continuous video frame called a "shot" in the minimum level of the tree data structure 615. 
Those shot-data structures contain two pointers. The 1st pointer in each shot-data structure points 
to the 1st frame of one shot in a video-data file, and the 2nd pointer directs the final frame of the 



shot in the same video-data file. For example, concerning the shot-data structure 620, the 1st 
pointer 650 points to the 1st frame 655 of the shot contained in the video-data file 600, and the 
2nd pointer 660 directs the final frame 665 of the shot. In this example, when performing 
extension or edit which carries out trimming for a shot, the pointer of the video-data file which 
specifies beginning or the end of that shot is moved appropriately. 

[0024]The data structure of a high level is above the shot-data structures 620-645 rather than 
specifying organization of the production after edit. The production comprises two or more 
coordinated data structures which form a tree data structure. There is the production route 675 
showing the whole production in the topmost part of a tree. A production is divided into the 
smaller subunits 680, 685, and 690 one by one, and results in the shot-data structures 620-645 
which form the base of a tree structure. Since a compulsory hierarchy is not imposed in this 
invention, a tree structure presents different appearance for every project reflecting how a 
specific producer composes a project. 

[0025]Under the compression digital video files 600-610, there are "virtual film strip" 700 
showing the present sequence of a video data. A shot is continuously shown in order so that it 
may be prescribed by the tree data structure of a production, as a figure shows. In order to give 
cooperation of information, the pointers 705, 710, 715, 720, 725, and 730 are provided from the 
video files 600, 605, and 610 to the virtual film strip 700. 

[0026] However, the edit-on screen method of this invention is not limited to the compressed 
video data base shown here. For example, instead, probably, it will also be possible to change the 
database which comprises the SMTPE time code used by edit in order to specify a "yne point" 
and an "out point" about actual videotape or a film stock. In the above, multimedia information 
was observed and the method and device to edit were explained. The material and arrangement 
of a component of this invention are considered that it can carry out modification and change by 
a person skilled in the art, without deviating from the meaning of this invention. 
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CLAIMS 



[Claim(s)] 

[Claim l]In a method of displaying and editing multimedia information by a display screen, A 
process in which the 1st unit of multimedia information is displayed on said display screen; The 
1st unit of said multimedia information is adjoined at said display screen, A process, and the; 1st 
portion which display the 2nd unit of multimedia information, SUPURINDA which comprises 
the 2nd portion combined with said 1st portion, and the 3rd portion combined with said 1st 



portion The 1st unit of said multimedia information, Said 1st portion of a process which is 
located between the 2nd unit of said multimedia information, and is displayed on said display 
screen, and; SUPURINDA, A process to which said 2nd portion and said 3rd portion are moved 
along with the 1st unit and 2nd unit of said multimedia information; A process in which the 1st 
unit and 2nd unit of said multimedia information on said display screen are edited according to a 
motion of SUPURINDA is comprised, A method into which multimedia information is edited by 
motion of SUPURINDA on a display screen. 

[Claim 2] A device into which multimedia information is displayed and edited, comprising: 
A display which combines with a digital computer and the; aforementioned digital computer, and 
has cursor; It combines with said digital computer, The 1st unit of multimedia information 
displayed on a cursor controller which controls a position of said cursor in said display, and the; 
aforementioned display; to said display. The 2nd unit, and the; 1st portion of multimedia 
information that are adjoined and displayed on the 1st unit of said multimedia information. 
The 2nd portion combined with said 1st portion. 

Cursor as which it comprises the 3rd portion combined with said 1st portion, and said 1st 

portion, said 2nd portion, and said 3rd portion are displayed on said display screen. 

The 1st unit of said multimedia information [ in / it can move freely on a display screen with a 

cursor controller combined with said display screen, and / said display ]. 

SUPURINDA displayed between the 2nd unit of said multimedia information; the 1st unit of 

multimedia information as which said SUPURINDA was displayed by said cursor controller. 

An editing means which answers being moved along with the 2nd unit of multimedia 

information, and edits the 1st unit and 2nd unit of said multimedia information. 

[Claim 3]A device into which multimedia information is displayed and edited, comprising: 
A display which combines with a digital computer and the; aforementioned digital computer, and 
has cursor; It combines with said digital computer, A cursor controller which controls a position 
of said cursor in said display; a track showing multimedia information as which it is on said 
display and each is displayed by a time-line method of at least one multimedia information. 
Each is connected with one multimedia information track, With cursor and a cursor controller. A 
display which comprises each element of a scale which displays at least one SUPURINDA 
which can move freely, and time base for every; multimedia information track along a 
multimedia information track, and an editing means which performs edit of a multimedia 
information track according to a motion of; SUPURINDA. 
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TECHNICAL FIELD 



[Industrial ApplicationJEspecially this invention relates to the method and device which provide 
a user with the intuitive method for editing the multimedia information which displays 
multimedia information on a display screen, and a display screen is shown about the field of a 
multimedia display and an editing device. 
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DESCRIPTION OF DRAWINGS 



[Brief Description of the Drawings] 

[Drawing ll The figure of the multimedia display editing system containing a player, a 
storyboard, and a strip viewer indicator. 

[Drawing 21 The figure showing one of the usable layered structures about a multimedia 
production. 

[Drawing 31 The figure showing one example of a strip viewer indicator. 

[Drawing 41 The figure showing "SUPURINDA" between the video frames belonging to two 

different shots in detail. 

[Drawing 51 The figure showing SUPURINDA used in order to edit the shot of video frame data. 

[Drawing 61 The video-data base created by this invention and the key map of a data structure. 

[Description of Notations] 

1, la, lb, and lc SUPURINDA 

1 1 Player 

13 Storyboard 

14 and 19 Caret 

15 Strip viewer 
17 Scroll bar 

21, 23, and 25 Media track 
27 Scale 

503 Head 

512 SUPURINDA 

504 Shaft part 

505 Crus-sinistrum-diaphragmatis part 
590 Cursor 

601 SUPURINDA 
603 Head 

605 Crus-sinistrum-diaphragmatis part 
607 Right trail part 
633,635 Frame 
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